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Abstract 


Predictive information warns the crew when a parameter approaches an alert 
range. This warning could increase the safety of flight because the added time before 
an alert range is reached may improve the crew's situation awareness . This warning 
may also decrease potential problems due to hardware failures by notifying the crew 
of a problem before hardware failure is reached . This experiment assessed certain 
issues about the usefulness of predictive information . The specific issues addressed 
were (I) the relative time criticality of failures, (2) the subjective utility of predictive 
information for different parameters or sensors , and (3) the preferred form and pre- 
diction time for displaying predictive information . To address these three issues, three 
separate tasks were administered to 22 airline pilots . These tasks were (I ) a checklist 
paired-comparison task, (2) a parameter-ordering task , and (3) a survey. As shown by 
the data , these pilots preferred predictive information on parameters they considered 
vital to the safety of the flight. These parameters were shown to be related to check- 
lists that pilots perform first. These pilots also preferred to know whether a parameter 
was changing abnormally and the time to a certain value being reached . In addition , 
they considered this information most useful during the cruise, the climb, and the 
descent phases of flight. Furthermore, these pilots preferred the information to pre- 
dict as far ahead as possible. 


Introduction 

Increasing the safety of flight primarily means mini- 
mizing the impact of human error because most accident 
investigations identify flight crew errors as a major 
causal factor (ref. 1, p* 265). One possible way of mini- 
mizing the impact of both human error and hardware fail- 
ure is to have predictive information available to the 
flight crew. The predictive information would aid the 
flight crew by indicating in advance when a parameter 
was moving away from its expected nominal value. Pre- 
dictive information may then increase the safety of flight 
by providing additional time for the crew to assess the 
situation, which may lead to more timely or appropriate 
responses for dealing with hardware failures. 

Predictive information may affect pilots’ situation 
awareness by several means. Because ‘‘humans. . . do not 
extract as much information from sources as they opti- 
mally should” (ref. 2, p. 63), especially during time criti- 
cal situations, predictive information may reduce time 
pressure because it would forewarn pilots of an alert 
message. Their attention could then be directed toward 
sources of relevant information. The easing of time pres- 
sure may decrease the number of errors “in reaction time 
tasks... [because people] tend to make more errors as 
they try to respond more rapidly” (ref. 3, p. 352). The 
added time also means the crew could move from tactical 
planning to more strategic planning. This possible shift is 
pertinent because the “importance of planning to overall 
mission effectiveness... [is the committing of] fewer 


operational errors among crews that made more contin- 
gency plans” (ref. 4, p. 157). Therefore, predictive infor- 
mation has the means to increase the flight crew’s 
situation awareness by allowing for more strategic plan- 
ning in situations that slowly develop. Predictive infor- 
mation also stops the practice of merely reacting to 
situations. 

A search through accident and incident reports 
revealed several failures where predictive information 
may have been beneficial. For example, a slow oil loss in 
the engines due to missing O-rings led to the Eastern Air- 
lines flight 855 accident (ref. 5). During the flight from 
Miami, Florida, to Nassau, indications of the failure were 
present 20 min into the flight when the flight crew had to 
shut down an engine due to low oil quantity. At this 
point, the flight was 50 min from Nassau. However, the 
crew was not able to ascertain the full problem until 
10 min later when all three engines read zero oil quantity. 
At this time, the crew elected to return to Miami because 
the weather at Nassau was deteriorating rapidly. The 
crew’s belief that the probability of having all three 
engines with zero oil quantity was very small also par- 
tially influenced the decision to return to Miami. The oil 
indications were not believed until 5 min later when 
another engine flamed out. Had the crew been alerted in 
a more salient manner that all engines were indeed losing 
oil, they may have diverted earlier. 

Detection of other fluid leaks, such as fuel, hydrau- 
lic, and pneumatic, may also readily benefit from 



predictive information. An Aviation Safety Reporting 
System (ASRS) database search found 131 incidents 
involving leaks that the flight crew did not detect before 
the parameter neared an alert range (ref. 6). In all of these 
incidents, information was available to the crew that the 
quantity was, in fact, decreasing. If the flight crew had 
detected these leaks before the alert range was reached, 
more time would have been available to plan efficiently 
for the best course of action. Detecting problems and 
notifying the crew before the alert range is reached is 
where predictive information may have the greatest ben- 
eficial impact. 

While predictive information has been hypothesized 
to be beneficial, these benefits have not been systemati- 
cally demonstrated or quantified. Thus, research is being 
conducted to substantiate and to quantify the benefits and 
the costs of predictive information. This study was the 
second conducted in a research program to look at the 
possibilities of incorporating predictive information into 
the flight deck. 

A previous study investigated the benefits of adding 
a predictive bug, which showed the value of a parameter 
5 sec in the future on a round dial display (ref. 7). Test 
subjects made long-term (more than 5 sec into the future) 
predictions using the dial display. That experiment found 
that pilots preferred the display with the predictive infor- 
mation. Pilots were more confident in their predictions 
and felt that they required less effort in making predic- 
tions. Contrary to these subjective results, the near-term 
predictive information hindered the pilot in making accu- 
rate long-term predictions when compared with having 
no additional information present. These results sug- 
gested that while predictive information may be benefi- 
cial, it must be in the proper form for safety to increase. 
In other words, the form of predictive information may 
need to be “more oriented to the user’s task” (ref. 8, p. 1) 
of expeditiously handling alerts and understanding their 
consequences on the flight. Therefore, some implementa- 
tion issues of predictive information must be studied 
before it can be fully decided whether predictive infor- 
mation will help pilots on the flight deck. 

Several issues, such as the ability to provide reliable 
and accurate predictive information, will affect the utility 
and the usability of the information. Although these 
issues are important, it is equally important to evaluate 
what pilots believe are the benefits of predictive informa- 
tion and what parameters they think should have 
predictive capabilities available. This combination of 
information is especially significant because “[p]ilots 
seem to simplify the decision-making task by focusing 
on only a few aspects of the information potentially 
available to them” (ref. 9, p. 179), Then, if predictive 
information is beneficial, specific issues, such as the ones 


listed earlier, relating to the implementation of predictive 
information on the flight deck may be studied. 

Experiment Objectives 

This experiment assessed certain issues about the 
usefulness of predictive information. The specific issues 
addressed were (1) the relative time criticality of failures, 
(2) the subjective utility of predictive information for 
different parameters or sensors, and (3) the preferred 
form and prediction time for displaying predictive 
information. The tasks that addressed these issues were 
(1) a checklist paired-comparison task, (2) a parameter- 
ordering task, and (3) a survey. 

Checklist Paired-Comparison Task Objective 

The objective of the checklist paired-comparison 
task was to find the time criticality of one failure relative 
to another. During multiple faults, a pilot may have to 
complete several checklists. Of specific interest was 
which checklist the pilot preferred to do first. 

Parameter-Ordering Task Objective 

The objective of the second task, the parameter- 
ordering task, was to learn what parameters or sensors, 
have the greatest potential gain from predictive informa- 
tion. Because many parameters on the flight deck could 
benefit from predictive information, pilot preferences on 
which parameters would benefit most were sought 
through a survey. 

Survey Objectives 

For the survey, the objectives were to begin to find 
(1) what form of predictive information pilots wanted on 
the flight deck, (2) where they wanted it, and (3) when 
they wanted it. To find the form of predictive informa- 
tion wanted, subjects considered four types of prediction, 
which ranged from a raw form of predictive information 
to a more processed form. The survey queried subjects 
about parameters that could possibly benefit from predic- 
tive information, such as system and flight control 
parameters. Lastly, subjects answered questions about 
which phases of flight the information would be useful 
for and how far into the future the system should predict. 

Experimental Variables 

A subset of HICAS (engine indicating and crew 
alerting system) messages obtained from a Boeing 767 
maintenance manual provided the checklists and the 
parameters used in the first two tasks (ref. 10, sect. 31- 
41-00, pp. 30-68). The parameters, the EICAS messages, 
and the checklists are all usually identified by the 
same name. The subset of EICAS messages contained 
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parameters that could take several minutes to reach an 
alert range. This time constraint was a consideration 
because of the possibility of applying predictive informa- 
tion to the parameters relevant to the checklists. 

Table 1 enumerates the parameters the experiment 
used. This table contains 1 1 advisories, 7 cautions, 
1 warning, and 3 parameters without EICAS messages, 
but the dials relating to these 3 parameters do have limits 
indicated. These three parameters, percentage of rota- 
tions per minute in the engines (N1 or N2), oil quantity 
(OIL QTY), and oil temperature (OIL TEMP), were not 
used in the checklist paired-comparison task. However, 
the parameter-ordering task used all 22 parameters listed 
in table 1 . 

The survey asked questions about the helpfulness of 
four predictive information types. These four types of 
information were a notification of (1) the value increas- 
ing or decreasing abnormally, (2) the rate of change, 
(3) the value of a parameter at a certain time in the future, 
and (4) the amount of time until a parameter reached a 
certain value. 

Experiment Design 

Subjects 

Twenty-two airline transport pilots participated in 
this experiment as subjects. Each subject was a current 
line pilot familiar with an EICAS type of alerting system. 
Four different commercial airlines were represented. In 
the survey results, an interaction was present between the 
airline the pilot was employed by and the type of infor- 
mation provided. Although statistically significant, the 
interaction was more likely an artifact of the small sam- 
ple size and the uneven sample size within the commer- 
cial airlines represented (10 from the first airline, 6 from 
the second airline, 4 from the third airline, and 2 from the 
fourth airline). 

The average age of the subjects was 43 years, rang- 
ing from 32 years to 53 years. They had an average of 
9638 flying hr, ranging from 2000 flying hr to 17 500 fly- 
ing hr. The average commercial experience for subjects 
was 13 years, with the most commercial experience 
being 28 years and the least, 2 years. An even split 
occurred between captains and first officers. Only one 
subject was female. 

Procedure 

When subjects first arrived, they received a verbal 
briefing on the purpose of research done at Langley 
Research Center and the general points of this experi- 
ment. (See fig. 1 for a schematic of the procedure.) Next, 
they completed the pilot background questionnaire 


(appendix A) and signed an informed consent form 
(appendix B). Afterwards, they read general pilot direc- 
tions (appendix C), which underscored certain points 
made during the verbal briefing. After the subjects said 
they understood everything and all their questions were 
answered, they received the checklist paired-comparison 
task directions. 

Checklist paired-comparison task procedure . Sub- 
jects did the checklist paired-comparison task first 
because the other two tasks included additional informa- 
tion; however, this task looked for an ordering irrespec- 
tive of any additional information. If subjects did not do 
this task first, they may have confused what information 
was available for it. 

The checklist paired-comparison task revealed pilot 
preferences in performing multiple checklists. Because 
the interest was in what action the pilot would take first 
and not the actual parameter and its associated warning 
message, the checklist titles associated with the alerts 
were used. 

These results, coupled with the results from the 
parameter-ordering task, aided in finding which parame- 
ters may benefit from predictive information, assuming 
pilots would want predictive information on the parame- 
ters that related to checklists they would perform for an 
alert message. It also quantified which failures pilots 
deemed were more critical relative to other failures. 

Subjects were first provided with the definition for 
each checklist (table 2). This ensured that all subjects 
were using the same basic definitions. Subjects had as 
much time as they wanted to look over these definitions. 
Once they said they understood the meaning of each 
checklist, the definitions were retrieved so that they did 
not use the list for ordering the checklists before and dur- 
ing the checklist paired-comparison task. Note that the 
checklists in this task did not refer to a particular sided- 
ness of a parameter, such as L ENG OIL PRESS. This 
factor was not addressed because the interest was in com- 
parisons between checklists, not within a particular 
checklist. 

Subjects then read the written directions for the 
checklist paired-comparison task (appendix D). The 
directions did not specify the flight phase so that subjects 
would be forced to use their own prioritization method. 

Once all questions about this task were answered, 
subjects did a few practice comparisons so that they 
could become comfortable with the format. A computer 
screen showed the names of two checklists. The subjects 
chose which one they would do first (fig. 2). Subjects 
could enter their preferences with either a mouse or a 
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trackball or through a keyboard. All subjects tried all 
methods and chose the method that best suited them. 

Each subject had a randomized ordering of pairs 
except for the constraint that the left and the right posi- 
tions of checklist pairs were counterbalanced across sub- 
jects to minimize any effects due to the ordering of the 
checklists on the computer screen. The computer 
recorded the subject’s preferences. 

Subjects were given the checklist procedures, which 
followed the Boeing format, for each alert message 
so that they could refer to them during the task 
(appendix E). The procedures were provided because the 
task was prioritization, not memorization. 

Parameter-ordering task procedure. The next task 
subjects did was the parameter-ordering task because it 
included fewer aspects of predictive information than the 
final task. Each subject ordered 22 index cards that had a 
parameter name on the front. On the back of each index 
card was a short description of the parameter. Table 3 
lists the parameters and the descriptions. The directions 
asked subjects to order the cards by which piece of infor- 
mation they most wanted to have predictive capabilities 
(appendix Fj. Note that predictive capabilities were 
defined as the ability to determine the future state. As 
before, the directions did not specify the flight phase so 
that subjects were forced to use their prioritization 
method. 

Survey procedure. Subjects completed the survey 
last because this task contained the most aspects of pre- 
dictive information (appendix G). The survey directions 
stressed to the subjects that the scales were continuous 
and they did not have to mark within a box. 

Dependent Measures 

Checklist paired-comparison task. For the checklist 
paired-comparison task, a subject compared each check- 
list title with the other 18 checklist titles. The comparison 
was by which checklist the subject would perform first. 
For a half-matrix, this resulted in 171 comparisons. 
These 171 comparisons were the dependent measures. 

Parameter-ordering task. For the parameter- 
ordering task, each subject ranked the 22 parameters (see 
table 3) with regard to which parameter can benefit most 
from predictive information. Thus, the rank each subject 
gave to each parameter was the dependent measure. 

Survey 

The data collected in this survey primarily consisted 
of subjective rankings and comments. Therefore, the 


dependent measures were the scaling of the information 
helpfulness and the subjects’ comments on their answers. 

Hypotheses 

Checklist paired-comparison task hypothesis. Glass 
cockpits with this type of alerting system do some order- 
ing of alert messages based on the time criticality of the 
problem. The faults are categorized into an advisory, 
caution, or warning (ref. 1 1). A warning is the most time 
critical event and it demands “immediate corrective or 
compensatory crew action” (ref. 11, p. 24). A caution 
requires “immediate crew awareness, and subsequent 
crew action” (ref. 1 1, p. 27). The least time critical event 
is an advisory, which gives “crew awareness, and may 
require subsequent or future crew action” (ref. 1 1, p. 27). 
The ordering within an alert category has been tradition- 
ally that the most recent alert is listed first (ref. 1 1, p. 60). 
Therefore, it was hypothesized that subjects would order 
the checklists primarily by alert level because this is the 
ordering method used by alerting systems. 

Ordering by subsystem was also considered because 
most checklists are indexed by subsystem. If there were a 
subsystem ordering, these results would begin to indicate 
which subsystems were considered by the subjects to be 
the most important when handling alerts. 

Parameter-ordering task hypothesis. This task 
studied the particular parameter for which predictive 
information would be provided. A previous study found 
that pilots wanted some form of predictive information 
available for engine instruments and “systems involving 
quantity, pressure, and temperature, as well as airspeed 
and altitude indicators” (ref. 7, p. 9). Because of this pre- 
vious research, it was hypothesized that subjects would 
rank engine parameters, parameters involving quantity, 
pressure, and temperature, and altitude indication high. 

The factor of subsystem could also influence the 
ordering because when one particular parameter begins 
to go out of tolerance, other parameters in the same sub- 
system are usually affected. These other parameters are 
often used by the flight crew as corroborating evidence 
of a problem. 

Survey hypothesis. The survey explored the predic- 
tive information pilots wanted. In a previous experiment, 
some pilots expressed a desire of knowing when an alert 
would occur over knowing the value of a parameter at a 
certain time in the future (ref. 7, p. 10). Thus, it was 
hypothesized that subjects would prefer the information 
detailing the time until a certain value is reached over the 
other types of information. Subjects would prefer this 
because it gives them information they can possibly use, 
such as when a parameter will exceed an alert threshold. 
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On the other hand, because “pilots process uncertainty in 
qualitative or linguistic terms rather than in numerical 
form,” (ref. 9, p. 185) a computationally intensive 
prediction, such as calculating a parameter value at a cer- 
tain time in the future, may not be required. Therefore, 
knowing that a value is moving unexpectedly may also 
be rated high. 

With regard to prediction span, previous research in 
this area found that any advanced warning of an alert was 
helpful, while a few pilots mentioned that “[t]he further 
into the future the better” (see ref. 7, p. 8). Therefore, it 
was hypothesized that subjects would want the informa- 
tion to predict ahead as far as possible. Early prediction 
may be especially desirable when the workload is low, 
such as during the cruise phase of flight, because more 
time is available to respond to the fault. 

Data Analysis 

Checklist paired-comparison task data analysis . 
The checklist paired-comparison task data were analyzed 
first by calculating the coefficient of consistence (ref. 12, 
p. 146). A coefficient of consistence close to 1.0 indi- 
cates that the subject was consistent in his rankings. This 
means that when the subject ranked A > B and B > C, he 
also ranked A > C in the majority of comparisons. A 
circular triad occurred if the subject ranked A > B and 
B > C, but C > A, which decreased the coefficient of 
consistence (ref. 12, p. 146). A chi-squared statistic 
determined whether the value of the observed coefficient 
of consistence was because subjects allotted their prefer- 
ences at random (ref. 12, p. 147). 

Computed next was the coefficient of agreement 
(ref. 12, pp. 148-149), which is similar to Kendall’s coef- 
ficient of concordance. This coefficient calculated the 
agreement among the subjects’ orderings.When the coef- 
ficient approaches 1 .0, the subjects have nearly equal 
orderings. Again, a chi-squared statistic determined 
whether the observed coefficient of agreement was 
because subjects allotted their preferences at random 
(ref. 12, pp. 152-153). 

The information collected was dominance data; that 
is, “the row object is preferred to, is chosen over, defeats, 
or otherwise dominates the column object” (ref. 13, 
p. 26). Therefore, PCPREF vl.O, a multidimensional 
analysis of preference data program for the personal 
computer, was also used (ref. 14). 

Parameter-ordering task data analysis . The data 
were analyzed with Friedman’s two-way analysis of vari- 
ance test. Kendall’s coefficient of concordance was also 
calculated to quantify the ordering agreement among 


subjects. The BMDP Statistical Software was used for 
these analyses (ref. 15). 

Survey data analysis . Average ratings were ana- 
lyzed in addition to subjects’ comments. An analysis of 
variance was performed with SPSS, where appropriate 
(ref. 16). A Newman-Keuls post hoc test was used to 
analyze multiple pairs of means for significant effects 
(ref. 17, pp. 346-351). (p < 0.05, where p is the propor- 
tion of test statistics smaller than observed, given the null 
hypothesis is true.) 

Results and Discussion 

Checklist Paired-Comparison Task Results and 

Discussion 

Subjects were internally consistent in which check- 
list they would do first. The average number of circular 
triads was 14 ± 10 (mean ± standard deviation) with a 
maximum of 44 and a minimum of 4 circular triads (table 
4). (The maximum number of triads possible was 285.) 
The average coefficient of consistence was 0.95 (table 4). 
Furthermore, no one subject chose preferences at random 
( p < 0.001). Subjects were also consistent among each 
other because the coefficient of agreement was 0.47. 
Again, the subjects were not random in their choices 
(p < 0.001). Therefore, subjects appeared to have a con- 
sistent method of determining which checklist they 
would perform first when compared with another check- 
list. They were also consistent among each other. 

Analysis from PCPREF further substantiated this. 
The solution space was one-dimensional. The ordering is 
shown in table 5. 

The ordering of which checklists subjects would per- 
form first was not strictly based on alert level, although it 
did seem to influence the ordering quite a bit as hypothe- 
sized (table 5). The ranking of the warning message 
comes after two caution messages and some caution mes- 
sages came after several advisory messages. Thus, sub- 
jects may be using prioritizing schemes other than the 
alert level, such as how crucial they thought the parame- 
ter was to the safety of flight. 

Parameter-Ordering Task Results and Discussion 

The results of Friedman’s two-way analysis of vari- 
ance were significant {p < 0.001). Also, Kendall’s coeffi- 
cient of concordance was 0.57; thus, there was agreement 
among subjects. Table 6 shows the average ranking. 

The prioritization of the alert messages showed that 
the highly ranked ones (altitude through fuel system 
pressure) were the messages that subjects most wanted to 
have predictive information. As hypothesized, these 
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contain parameters relevant to the engine (EGT, oil pres- 
sure, oil quantity, oil temperature, N1 and N2, fuel quan- 
tity, and fuel system pressure), and altitude indication, as 
well as the perceived cabin altitude as indicated by pres- 
sure. Thus, subjects most wanted predictive information 
relating to engine parameters (to keep the engines 
healthy), altitude (to keep the plane healthy), and cabin 
altitude (to keep the passengers healthy). 

Because the parameter-ordering task contained 
parameters directly related to the checklists seen in the 
first task, it was of interest to see whether subjects 
wanted predictive information in the same areas where 
they would do the checklist first for that alert message. 
Therefore, both orderings were compared. First, any 
parameters or checklists not having a direct partner were 
eliminated. Both lists were then normalized around the 
remaining 18 parameters, or checklists. Table 7 shows 
the normalized rankings of the checklist and parameter 
orderings. The correlation between the rankings of the 
normalized checklist paired-comparison task and the 
parameter-ordering task was 0.95. Therefore, subjects 
seemed to most want predictive information on the 
parameters relating to checklists they considered the 
most important. 

Survey Results and Discussion 

Type off predictive information wanted. When asked 
in general which type of predictive information they 
wanted, subjects responded that they wanted to know 
whether a value was increasing or decreasing abnormally 
and the amount of time until a certain value was reached 
(table 8). These two types of information were not signif- 
icantly different from one another. 

Subjects did want to control the value to which the 
information was predicted. Two-thirds of the subjects 
desired control over it because they wanted to use their 
safety margin and priority. The other two types of predic- 
tive information, the rate of change and the value at a 
certain time, were wanted the least by subjects and were 
not significantly different from each other. This prefer- 
ence changed slightly depending on what other aspect 
subjects were considering for the question (table 8). The 
most noticeable change came when subjects were also 
considering the type of predictive information they 
wanted for each phase of flight. In this case, they prima- 
rily wanted to know whether the value was moving 
abnormally. This type of predictive information would 
allow the crew to be cognizant of an abnormally moving 
parameter with the ability to calculate the time to a cer- 
tain value if their workload permitted this. Overall, sub- 
jects wanted two types of predictive information, 
whether a parameter was moving abnormally and the 
time when a certain value would be reached. 


Prediction time. An interaction was present between 
the type of predictive information subjects wanted and 
how far into the future subjects wanted the information to 
predict. As shown in figure 3, subjects wanted to know at 
all times whether a value was changing abnormally. As 
one subject put it, knowing that a value is changing 
abnormally is a “wake-up call.” The amount of time to 
reach an alert did not matter in this case. Subjects also 
rated highly knowing the rate of change because of its 
ability to inform them on how fast conditions are 
changing. 

Subjects wanted the time to a value and the value at a 
certain time when they had the most time to troubleshoot 
a problem. These ratings increased as pilots had more 
time to troubleshoot a problem. Subjects wanted the time 
to a value slightly more than they wanted the value at a 
certain time. Thus, knowing a value is moving abnor- 
mally and the rate of change are not very dependent on 
the amount of time to reach an alert. The time to reach a 
value is more desirable; however, the information is 
more beneficial the more time the subject has until an 
alert. 

Phase of flight and predictive information. Sub- 
jects wanted predictive information the most during the 
cruise, the climb, and the descent phases of flight 
(table 8). A few subjects specifically mentioned that they 
did not want the information during the critical phases of 
take-off and landing. 

Situation awareness. Subjects thought that know- 
ing whether a value was increasing or decreasing abnor- 
mally would increase situation awareness the most 
(table 9). This was primarily because it would direct pilot 
attention to the critical parameter earlier. Six subjects 
mentioned this specifically. Ratings for the other three 
types of information were grouped together, but ratings 
for time to a certain value had the next highest average. 

Discussion. The hypothesis that pilots most wanted 
predictive information detailing the time until a certain 
value is reached was partly confirmed. Subjects preferred 
to know whether a parameter was changing abnormally 
and the time to a certain value. Notice that these two 
types of information are on opposite ends of the scale 
related to processing. It appears that these subjects 
wanted to know whether a parameter was changing 
abnormally so that a quick decision could be made on 
whether the degradation warranted more attention. Once 
they determined that the problem demanded some atten- 
tion and workload permitted, subjects wanted the time to 
a certain value so that they knew how long they had to 
troubleshoot. This was supported by the result that 
subjects wanted the information in phases of flight 
where they had the most time to respond to an oncoming 
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fault — cruise, climb, and descent. Furthermore, these 
subjects said they wanted predictive information as far 
ahead as possible, which would give them more time to 
troubleshoot and resolve the problem before it became an 
emergency. 

Concluding Remarks 

The pilots in this experiment preferred predictive 
information on parameters they considered vital to the 
safety of flight, such as altitude, engine parameters, and 
cabin altitude. These parameters were shown to be 
directly related to the checklists that pilots would per- 
form first for an alert message. Although the established 
alert structure is based on time to respond, the high corre- 
lation in the subjects’ ordering of alert messages indi- 
cates that the current ordering based on time to respond 
may not fully take into account all the factors pilots 
consider. 

One factor that these pilots did seem to consider was 
aircraft subsystems. The top rated checklists covered 


most of the aircraft’s subsystems: (1) altitude deals with 
aircraft position, (2) engine overheat and engine oil pres- 
sure encompass the engine and oil systems, (3) cabin alti- 
tude covers the pneumatics and environmental system, 
(4) low fuel and fuel system pressure are related to the 
fuel system, and (5) flap/slat asymmetry entails the 
hydraulic and controls system. 

Other results from this experiment suggest that 
knowing whether a parameter is changing abnormally 
and the time until it reaches a certain value has the poten- 
tial to benefit the safety of flight. This may especially be 
true if the flight crew has time to work on the problem 
before it becomes an emergency. One such time is during 
cruise, especially when the parameters are considered 
basic to the safety of flight. 


NASA Langley Research Center 
Hampton, VA 23681-0001 
November?, 1995 


7 



Date Administered: k Administrator: t Study: A Subject #: 

Appendix A 


NASA Langley Flight Management Division 
Pilot Background Questionnaire 


1. General Information 

Full Name: 

First, Middle, Last 

Address: 

Street and Number, or P.O. Box 


City, State, Zip Code, and Country (if not USA) 

Work Phone: ( ) 

Area Code Number 


Home Phone: ( ) 

Area Code Number 


Birth Date: 

Month/Day /Year 


Do you wear corrective lenses when you fly? 




2. General Experience Information 

Current/Most Recent Airline: 

Current/Most Recent Position: 

Captain, First Officer, Engineer, etc. 


Are you currently flying military? 



No 


□ 


Years Flying Commercial (approximate): 

Years Flying Military (approximate): 

Total Hours Flying (approximate): 

Total Hours Flying as Pilot-in-Command (approximate): 

Years of formal education: (e.g. high school graduate =12) 
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Date Administered: 


Administrator: 


Study: 


Subject#: 


3. Specific Aircraft Experience Information 

Please list the types of aircraft on which you have experience, beginning with the most recently 

flown. 

For each aircraft, please check the columns to indicate your 
approximate number of hours flying experience, and 
approximate number of hours simulator experience. 

If you were an Instructor (I) or a Check Airman (CA) on any of these 

aircraft, please indicate by checking the column entitled "I/CA". 

If you are currently type rated on any of these aircraft, please indicate by 
checking the last column. 


Aircraft Type 

Hours in Type 

Simulator Hours 

I/CA 

Currently 
Type Rated? 


<300 

300-1000 

> 1000 

0 

<50 

>50 

. . . , 










































































Please check the appropriate column to indicate the approximate number 

of years of experience you have for each of the following categories: 


1 Specific Aeronautical Experience 

Years Experience | 





i <i ii'- 5 

>5 

Long-range, Over-water (Class II) Operations ( 2 engines) 




Long-range, Over-water (Class II) Operations ( > 2 engines) 




Total Multi-Engine (Captain or F/O, Military or Civil) 




Glass Cockpit (i.e. EFIS/CRT or FMS) 
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Date Administered: 


Administrator: . Study: 


Subject #: 


4. Previous and Future Experience as a NASA Subject 


Have you ever participated in a NASA research project? Yes 


□ 


No 


□ 


If "Yes", please briefly describe the test(s)/ interview(s) and, if possible, 
give the names of the researchers: 


Would you want to participate in future NASA experiments? Yes 


Qno 2 


If "Yes”, please indicate in which of the following types of 
experiments you would be interested: 


1. 

Tests which require flying the NASA B737? 
(requires 737 rating) 

Yes Q 

No 

2. 

Tests which require flying one of the NASA 
simulation facilities? 

Yes Cjl 

No ^ 

3. 

Evaluations of new displays and flight deck 
systems? (no flying involved) 

Yes Q 

No d 

4. 

Interviews and studies relating to aircraft 
safety, automation, etc.? 

Yes Ql 

No Q 


Approximately how much lead time (days or weeks) would you require 
for scheduling appointments? 
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Date Administered : , Administrator: » Study: 4 Subject #: 

5. Returning the Questionnaire 

Thank you for completing this questionnaire. This information will be used to help 
us understand results in terms of pilot characteristics and to schedule you for future 
participation if you are interested. All information contained herein will be kept 
confidential. If you have any additional information you think would be useful, please feel 
free to write on the back of these forms. 

If you have any questions please call Ms. Anna Trujillo at 

804-864-8047 (EST). 


Please complete this questionnaire and bring it with you to the experiment. 
Thank you. 
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Appendix B 


Informed Consent Form 

I understand the general purpose of the investigation as described and my duties associated with 
the experiment. I have been briefed on the expected duration and scheduling of my participation 
in the experiment. I realize that performance and subjective data will be collected as I perform 
the tasks in the experiment. I understand that all data resulting from participation in this 
experiment will be held confidentially by the experimenters, will be referenced only by subject 
number, and that it will be summarized to assure my anonymity. My participation in this 
experiment is voluntary and I understand that I may withdraw from the experiment at any time 
without penalty. 


(signature) 


(date) 


(please print name) 


Anna Trujillo 
Principal Investigator 


Thank you for your participation in this study. 
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Appendix C 


General Pilot Directions 

The objective of this study is to improve our understanding of the importance of various 
parameters and how possibly new information will affect their importance. You will be doing 2 
card ordering type tasks and completing a survey, which will help our understanding of your 
needs. 


The experiment is not aimed at assessing you as an individual, but rather, is 
aimed at assessing the importance of information to you in general. The data that we 
collect during this experiment will be confidential and used only for scientific purposes. 

The experiment will take about 3 hours to complete. We appreciate your participation. 
It is critical to the integrity of the experiment that pilots do not know the specifics of the tasks 
they will be doing beforehand, so please, do not discuss the experiment with other pilots. 
The experimenter will answer any questions you have. 



Appendix D 


Checklist Paired-Comparison Directions 

On the screen in front of you, two checklist titles will be shown. Please highlight with the mouse 
or trackball and double click on the checklist you would perform first if both alerts occurred at the 
same time. If you do not want to use the mouse or trackball, you may use the left and right 
arrow keys to highlight your choice. You must then press the Enter key to enter your choice. 

You will have five practice comparisons so that you can become familiar with the task and 
comfortable with the input keys. Take as long as you need and I will answer any general 
questions you have. Thank you. 
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Appendix E 


Checklists 

Message : ALTITUDE ALERT 

Crew awareness. 

Message : APU BAT DISCH 

Crew awareness. 

Message : APU BTL 

Crew awareness. 
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Message : CABIN ALTITUDE 

ISOLATION VALVE SWITCHES OFF 

If cabin altitude cannot be controlled: 

PASSENGER OXYGEN ON 

DESCENT ACCOMPLISH 


Without delay, close thrust levers, extend speedbrakes and descend at Vmo/Mmo. 
Level off at lowest safe altitude or 10,000 feet, whichever is higher. 

If structural integrity is in doubt, limit airspeed and avoid high maneuvering loads. 
If cabin altitude can be controlled and both duct pressures remain normal: 


PACK CONTROL SELECTOR OFF 

If cabin altitude can be controlled and one duct pressure remains low: 
ENGINE BLEED AIR SWITCHES 

(Affected side) OFF 

BLEED OFF messages are displayed 
ISOLATION VALVE SWITCH 

(Normal side) ON 

PACK CONTROL SELECTOR 

(Affected side) OFF 

HYDRAULIC DEMAND PUMP 

(Affected side) OFF 

HYD PRESS DEM message displayed. 

Do not use wing anti-ice. 


Sufficient bleed air may not be available for nacelle anti-ice if N1 is less than 70% above 
10,000 feet or less than 55% below 10,000 feet. 

LANDING PREPARATION: 

Allow time during approach for secondary flap operation. 

PACK CONTROL SELECTORS SET 

Maximum one pack on. 

FLAPS EXTEND OR RETRACT AS REQUIRED 

During flap operation the FLAPS PRIMARY message is displayed. 

Message : ENG OVHT 

THRUST LEVER RETARD 

Retard thrust lever slowly until message is no longer displayed. If message stays on with 
thrust lever fully retarded: 

THRUST LEVER CLOSE 

FUEL CONTROL SWITCH CUTOFF 
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Message : ENG BTL 

Crew awareness. 


Message : FLAP/SLAT ASYM 

AUTOPILOT DISENGAGE 

Higher than normal control column force may be required to prevent unwanted roll. 

Message : FLAP/SLAT DISAGREE 

AUTOPILOT DISENGAGE 

FLAPS 0° 

Higher than normal control column force may be required to prevent unwanted roll. 
AUTOPILOT ENGAGE 

Message : FUEL CONFIG 

Configure fuel pumps and crossfeed valves as required to balance fuel. When fuel is balanced, 
return to normal fuel system configuration. 

Message : FUEL PUMP PRESS 


FUEL PUMP 

(Affected System) OFF 

BOOST PUMP 

(Affected System) ON 


Message : FUEL SYS PRESS 

FUEL PUMPS 

(Affected System) ON 

Avoid high nose up attitude and excessive acceleration. 


Message : GEN DRIVE 

GENERATOR DRIVE DISCONNECT SWITCH PUSH 

DRIVE DISC and ELEC GEN OFF messages are displayed. 
APU (If Available) START 



Message: HYD PUMP OVHT 


HYD PUMP SWITCH 

(Affected Pump) OFF 

When HYD PUMP OVHT message is no longer displayed: 

HYD PUMP SWITCH 

(Affected Pump) ON 

If HYD PUMP OVHT message is displayed again: 

HYD PUMP SWITCH 

(Affected Pump) OFF 

Message : HYD PUMP PRESS 

HYD PUMP SWITCH 

(Affected Pump) OFF 


Message : HYD QTY 

HYD PUMP SWITCHES 

(Affected System) OFF 

Note inoperative items. 

Complete Landing Preparation. 

Message : HYD SYS PRESS 

DEMAND PUMP SELECTOR 


(Affected System) ON 

ENGINE PUMP SWITCH 

(Affected System) OFF 

If HYD PRESS SYS message remains displayed: 

DEMAND PUMP SWITCH 

(Affected System) OFF 

Note inoperative items. 

Complete Landing Preparation. 

Messag e: LOW FUEL 

CROSSFEED VALVE SWITCHES (All) ON 

MAIN PUMP SWITCHES (All) ON 


Avoid high nose up attitude and excessive acceleration. 

Message : MAIN BAT DISCH 

Crew awareness. 
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Message : N1 or N2 

THRUST LEVER 

Affected System) RETARD 

If N1 or N2 does not decrease below the red line limit or remains in amber band for longer than 
10 minutes: 

THRUST LEVER CLOSE 

FUEL CONTROL SWITCH CUTOFF 

Message : OIL PRESS 

OIL PRESSURE INDICATION 

If oil pressure at or below red line limit 

THRUST LEVER 

FUEL CONTROL SWITCH 

Message : OIL QTY 

OIL PRESSURE INDICATION 

If oil pressure at or below red line limit 

THRUST LEVER 

FUEL CONTROL SWITCH 

Message : OIL TEMP 

THRUST LEVER ADVANCE TO MID POSITION 

If temperature does not decrease below the red line limit or remains in amber band for longer 
than 20 minutes: 

THRUST LEVER CLOSE 

FUEL CONTROL SWITCH CUTOFF 


. CHECK 

..CLOSE 

CUTOFF 


. CHECK 

.. CLOSE 
CUTOFF 
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Appendix F 


Parameter Ordering Task Directions 

Each of the 22 cards contains information currently available on the flight deck. Please order the 
cards by how useful you think predictive information would be on that piece of information. 
Predictive capabilities are the ability to calculate the future state of something. You should finish 
with one pile. The first card should be where you would want predictive capabilities the most 
while the last card should be where you want predictive capabilities the least. Take as long as 
you need. I will answer any general questions you have. Thank you. 
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Appendix G 
Survey 


1 . Occupation 

(e.g., pilot, airframer, airline, human factors engineer) 

2. Do you have a pilots license? □ □ If Yes, how many hours? 

Yes No Type rating 

Check the types of aircraft you have been qualified to fly. 


Aircraft Type 

Qualified? 

single engine 


two-engine private 


corporate jet 


turboprop 


commercial transport jet 


military transport 


military fighter 


helicopter 



For each of the following questions, please either write out your answer or mark on the 
scale the location that best describes your answer. The areas between the extremes on the scale 
indicate not as much and the dividers are for anchoring purposes. If you run out of room for your 
written answers, feel free to use the backs of the sheets. 

Definitions: Not Helpful - will not aid the flight crew and the flight 

(safety, passenger comfort, etc.) 

Very Helpful - will greatly aid the flight crew and the flight. 


Decrease Workload - workload would decrease from the present level 
Increase Workload - workload would increase from the present level 


Decrease Situation Awareness - situation awareness would decrease from its present 

level 

Increase Situation Awareness - situation awareness would increase from its present 

level 


Little Training - a little time and practice required 
Much Training - a great deal of time and practice required 

Note: When examples are given, they are generic so as not to bias your response. 




3. With respect to an airplane’s system, how helpful would it be to know: 

(1) A parameter’s value is increasing or decreasing abnormally? 

(e.g., temperature is rising faster than expected) 


Not Very 

Helpful Helpful 


(2) A parameter’s rate of change (units/time)? 
(e.g., quantity is decreasing at 50 units/hour) 


Not Very 

Helpful Helpful 


(3) A parameter’s value at a specific time in the future? 
(e.g., pressure will be at 1 psi in 30 minutes) 


Not Very 

Helpful Helpful 


Should the operator have control over the look-ahead time? U U 

(e.g., forecasting 2 minutes ahead vs 2 hours ahead) Yes No 

Why? 


(4) the amount of time until a specific value is reached? 
(e.g., in 2 hours, an overheat condition will be present) 


Not Very 

Helpful Helpful 


Should the operator have control over the specific value? 
Why? 
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For the following rating questions, please rate using each type of parameter information 
presented in question 3. These types were (1) a parameter’s abnormal direction of movement, 
increasing or decreasing, (2) a parameter’s rate of change, (3) a parameter’s value at a specific 
time in the future, and (4) the amount of time until a specific value is reached. Please see the 
example below for further clarification. 

At the end of each question, space is provided so that you may comment on the question. 
If not enough space is present, use the back of the sheet. 


A. Question about helpfulness of the four types of information (m, r, q, t)? 
m = value increasing or decreasing abnormally (moving) 
r = rate of change 

q = value at some point in the future (quantity) 
t = amount of time until a certain value is reached 



k 



r: 




1 / 1 



1 Lx 



t: 




k 1 

! 

Not 


Very 


Not 



Very 

Helpful 

Helpful 

Helpful 


Helpful 


Comments: 



4. How helpful would each of the four types of information (m, r, q, t) be for: 
m = value increasing or decreasing abnormally (moving) 
r = rate of change 

q = value at some point in the future (quantity) 
t = amount of time until a certain value is reached 


(1) System Parameters (e.g., hydraulic quantity, egt)? 


m: 




q: 1 1 1 1 1 

t: 

1 1 l 

1 1 1 

Not Very 

Helpful Helpful 


Not 

Helpful 

Very 

Helpful 

(2) Navigation (e.g., future position, time to a location)? 




m: 1 1 | 

r: 

1 1 1 1 

1 1 1 

q: 1 1 1 1 1 1 

t: 

i i i 

! | | 

Not Very 

Helpful Helpful 


Not 

Helpful 

Very 

Helpful 

(3) Flight Control (e.g., altitude, speed)? 




m: | | ! | | 

r: 

1 1 1 

1 1 1 

q: 1 1 1 1 1 1 

t: 

i i i i i i 

Not Very 

Helpful Helpful 


Not 

Helpful 

Very 

Helpful 

(4) Fault Development (e.g., engine failure, cargo fire)? 




m: | | | | | 

r: 

i i i 


. 11 , 11 , 

t: 

i i i 


Not Very 

Helpful Helpful 


Not 

Helpful 

Very 

Helpful 

(5) Other 


? 


m: | | | | | | 

r: 

1 1 1 


q: 1 1 i i 1 1 

t: 

i i i 


Not Very 

Helpful Helpful 


Not 

Helpful 

Very 

Helpful 
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Comments regarding Question 4 
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5. How helpful would each of the four types of information (m, r, q, t) be during: 
m = value increasing or decreasing abnormally (moving) 
r = rate of change 

q = value at some point in the future (quantity) 
t = amount of time until a certain value is reached 


(1 ) Taxi? 


m: 



r: 



q: 



! 

t: 

! , 

I,,, 


Not 

Very 


Not 

Very 


Helpful 

Helpful 


Helpful 

Helpful 

(2) Takeoff? 






m: | 

1 1 1 1 

| 

r: 

i i 

1 1 1 1 

q: 



i 

t: 

i i 

I,,, 


Not 

Very 


Not 

Very 


Helpful 

Helpful 


Helpful 

Helpful 

(3) Climb? 






m: 

1 1 1 1 

...i 

r: 

i i 

Mil 

q: 




t: 




Not 

Very 


Not 

Very 


Helpful 

Helpful 


Helpful 

Helpful 

(4) Cruise? 






m: 

1 1 1 1 1 

| 

r: 

i i 

till 

q: 

I,,, 

i | 

t: 

i i 

Mil 


Not 

Very 


Not 

Very 


Helpful 

Helpful 


Helpful 

Helpful 

(5) Descent? 






m: 

till 1 

| 

r: 

i i 

Mil 

q: 

I,,, 

! | 

t: 


1111 


Not 

Very 


Not 

Very 


Helpful 

Helpful 


Helpful 

Helpful 
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Question 5 Continued: 

6. How helpful would each of the four types of information (m, r, q, t) be during: 
m = value increasing or decreasing abnormally (moving) 
r = rate of change 

q = value at some point in the future (quantity) 
t = amount of time until a certain value is reached 

(6) Approach? 



Helpful Helpful Helpful Helpful 

(7) Landing? 



Helpful Helpful Helpful Helpful 

Comments: 
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7. How helpful would each of the four types of information (m, r, q, t) be during: 
m = value increasing or decreasing abnormally (moving) 
r = rate of change 

q = value at some point in the future (quantity) 
t = amount of time until a certain value is reached 

(1) Time-critical situations (<5 seconds)? 



Helpful Helpful Helpful Helpful 

(2) Time-pressured situations (5 - 60 seconds)? 




Not Very Not Very 

Helpful Helpful Helpful Helpful 

(3) Time-compressed situations (1-5 minutes)? 



Helpful Helpful Helpful Helpful 

(4) Slowing evolving conditions (5-15 minutes)? 



Helpful Helpful Helpful Helpful 

(5) Very slowly evolving conditions (>15 minutes)? 



Helpful Helpful Helpful Helpful 
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Comments regarding Question 6 



8. How helpful would each of the four types of information (m, r, q, t) be for: 
m = value increasing or decreasing abnormally (moving) 
r = rate of change 

q = value at some point in the future (quantity) 
t = amount of time until a certain value is reached 


(1) Low speed aircraft? 


m: 

r: 



q: 1 1 1 I 1 1 

t: 



! 

Not Very 

Helpful Helpful 


Not 

Helpful 



Very 

Helpful 

(2) Corporate jets? 






m: | | | | 

r: 


| 

j i 


q: 1 1 1 III 

t: 

I,,, 

i 

Not Very 

Helpful Helpful 


Not 

Helpful 



Very 

Helpful 

(3) Commercial jets? 






m: | | j j 

r: 

| | 

| 

i i 


q: 1 1 1 1 

t: 

i | 

i 

i i 

! | 

Not Very 

Helpful Helpful 


Not 

Helpful 



Very 

Helpful 

(4) High speed aircraft (>Mach 1)? 






m: | | | 

r: 

1 L 

| 

j i 

| 

q: 1 III 1 

t: 



! | 

Not Very 

Helpful Helpful 


Not 

Helpful 



Very 

Helpful 


Comments: 
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9. In what situations would this information be the most beneficial and why? 


Value increasing or decreasing abnormally 


Rate if change 


Value at some point in the future 


Amount of time until a certain value is reached 
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10. How would each of the four types of information (m, r, q, t) change the flight crew’s 
workload? 

m = value increasing or decreasing abnormally (moving) 
r = rate of change 

q = value at some point in the future (quantity) 
t = amount of time until a certain value is reached 


m: 




q= 






Increase Decrease 

Workload Workload 


Increase Decrease 

Workload Workload 


Why? 


11. How would each of the four types of information (m, r, q, t) change the flight crew’s situation 
awareness? 

m = value increasing or decreasing abnormally (moving) 
r = rate of change 

q = value at some point in the future (quantity) 
t = amount of time until a certain value is reached 


q: 






Decrease 

Situation 

Awareness 


Increase 

Situation 

Awareness 


Decrease 

Situation 

Awareness 


Increase 

Situation 

Awareness 


Why? 


32 



Any other comments 


Thank you for taking the time to fill out this survey. Please mail the completed survey to 


Anna Trujillo 

NASA Langley Research Center 
MS 152 

Hampton, VA 23681-0001 
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Table 1. Parameters Used in Experiment 


Parameter/EICAS message 

Alert level 
(a) 

ALTITUDE ALERT 

Caution 

APU BAT DISCH 

Advisory 

APU BTL 

Advisory 

CABIN ALTITUDE 

Warning 

EGT b 

Caution 

ENG BTL 

Advisory 

FLAP/SLAT ASYM 

Caution 

FLAP/SLAT DISAGREE 

Caution 

FUEL CONFIG 

Advisory 

FUEL PUMP PRESS 

Advisory 

FUEL SYS PRESS 

Caution 

GEN DRIVE 

Advisory 

HYD PUMP OVHT 

Advisory 

HYD PUMP PRESS 

Advisory 

HYD QTY 

Advisory 

HYD SYS PRESS 

Caution 

LOW FUEL 

Caution 

MAIN BAT DISCH 

Advisory 

Nl,2 


OIL PRESS 

Advisory 

OIL QTY 


OIL TEMP 



a Alert level applies to EICAS message of same name. 
b EICAS message for EGT is ENG OVHT. 




Table 2. Definition of Checklists 


Checklist title 

Checklist procedure followed when — 

ALTITUDE ALERT 

Deviation from selected altitude by ±300 ft 

APU BAT DISCH 

APU battery discharging above limit 

APUBTL 

APU fire bottle pressure below limit 

CABIN ALTITUDE 

Cabin altitude above 10000 ft 

ENG BTL 

Engine fire bottle pressure below limit 

ENG OIL PRESS 

Engine oil pressure below limit 

ENG OVHT 

Engine over-temperature 

FLAP/SLAT ASYM 

Difference between banks of flap/slat above limit 

FLAP/SLAT DISAGREE 

Difference between flap/slat actual and commanded position above limit 

FUEL CONFIG 

Fuel tank fuel quantities differ above limit 

FUEL PUMP PRESS 

Fuel pump pressure below limit 

FUEL SYS PRESS 

Fuel system pressure below limit 

GEN DRIVE 

Generator drive oil press below limit or oil temperature above limit 

HYD PUMP OVHT 

Hydraulic pump temperature above limit 

HYD PUMP PRESS 

Hydraulic pump pressure below limit 

HYDQTY 

Hydraulic fluid quantity below limit 

HYD SYS PRESS 

Hydraulic system pressure below limit 

LOW FUEL 

Fuel quantity in main fuel tanks below limit 

MAIN BAT DISCH 

Main battery discharging above limit 
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Table 3. Parameters for Parameter-Ordering Task 


Parameter 

Description 

ALTITUDE 

Altitude of aircraft 

APU BAT AMP 

APU battery amperage 

APU BTL PRESS 

APU fire bottle pressure 

CABIN ALTITUDE 

Cabin altitude of aircraft 

EGT 

Exhaust gas temperature of engine 

ENG BTL PRESS 

Engine fire bottle pressure 

FLAP/SLAT ASYMMETRY 

Position difference between banks of flaps/slats 

FLAP/SLAT DISAGREE 

Difference between flap/slat actual and commanded position 

FUEL CONFIG 

Fuel quantity difference between fuel tanks 

FUEL PUMP PRESS 

Output pressure of a fuel pump 

FUEL QTY 

Fuel quantity in fuel tanks 

FUEL SYS PRESS 

Fuel system pressure in fuel system 

GEN DRIVE PRESS or TEMP 

Generator drive pressure or temperature 

HYD QTY 

Hydraulic fluid quantity in hydraulic system 

HYD PUMP PRESS 

Output pressure of a hydraulic pump 

HYD PUMP TEMP 

Hydraulic pump temperature 

HYD SYS PRESS 

Hydraulic system pressure in hydraulic system 

MAIN BAT AMP 

Main battery amperage 

N1 or N2 

N1 or N2 percentage 

OIL PRESS 

Oil pressure in engine 

OIL QTY 

Oil quantity in engine 

OIL TEMP 

Oil temperature in engine 
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Subject 

Number of circular triads 

Coefficient of consistence 

1 

26 

0.909 

2 

14 

0.951 

3 

4 

0.986 

4 

6 

0.979 

5 

5 

0.982 

6 

16 

0.943 

7 

17 

0.940 

8 

7 

0.975 

9 

4 

0.986 

10 

7 

0.975 

11 

24 

0.916 

12 

6 

0.979 

13 

20 

0.930 

14 

23 

0.919 

15 

23 

0.919 

16 

8 

0.972 

17 

10 

0.965 

18 

44 

0.846 

19 

11 

0.961 

20 

4 

0.986 

21 

20 

0.923 

22 

11 

0.961 

Average 

14.09 

0.950 





Table 5. One-Dimensional Vector Space Ordering From 
PCPREF for Checklist Paired-Comparison Task 


Checklist 

Magnitude 

Alert level 

ALT 

0.361 

Caution 

ENG OVHT 

0.340 

Caution 

CAB ALT 

0.297 

Warning 

LOW FUEL 

0.253 

Caution 

ENG OIL PRESS 

0.183 

Advisory 

FUEL SYS PRESS 

0.147 

Caution 

FLAP/SLAT ASYM 

0.125 

Caution 

HYD PUMP OVHT 

0.017 

Advisory 

FUEL PUMP PRESS 

-0.001 

Advisory 

FLAP/SLAT DIS 

-0.002 

Caution 

HYD QTY 

-0.033 

Advisory 

GEN DRIVE 

-0.039 

Advisory 

HYD SYS PRESS 

-0.039 

Caution 

FUEL CONFIG 

- 0.072 

Advisory 

HYD PUMP PRESS 

-0.152 

Advisory 

MAIN BAT 

-0.244 

Advisory 

ENG BTL 

-0.330 

Advisory 

APU BAT 

-0.375 

Advisory 

APU BTL 

-0.432 

Advisory 




Table 6. Parameter-Ordering Task Average Rankings 


Checklist 

Average ranking 
(a) 

Alert level 

ALTITUDE 

2.7 

Caution 

FUEL QTY b 

5.4 

Caution 

EGT b 

6.1 

Caution 

CABIN ALTITUDE 

6.7 

Warning 

OIL PRESS b 

7.4 

Advisory 

OIL QTY b 

7.6 

Advisory 

N1 or N2 b 

8.3 

Advisory 

OIL TEMP b 

8.8 

Advisory 

FUEL SYS PRESS b 

10.0 

Caution 

HYD QTY b 

10.3 

Advisory 

FLAP/SLAT ASYMM b 

10.8 

Advisory 

FUEL CONFIG” 

11.2 

Advisory 

HYD SYS PRESS C 

11.4 

Caution 

GEN DRIVE C 

12.7 

Advisory 

FLAP/SLAT DISAGREE 0 

13.5 

Advisory 

HYD PUMP TEMP C 

13.8 

Advisory 

FUEL PUMP PRESS 0 

14.0 

Advisory 

HYD PUMP PRESS 0 

14.5 

Advisory 

MAIN BAT 1 

17.4 

Advisory 

ENG BTL d 

19.4 

Advisory 

APU BAT d 

19.5 

Advisory 

APU BTL d 

21.2 

Advisory 


a The lower the number, the higher ranking. 
Engine and propulsion system. 
c Flight controls. 
d Power. 




Table 7. Normalized Average Rankings of Checklist 
Paired-Comparison Task and Parameter-Ordering Task 


Checklist/parameter 

Average ranking for — 

(a) 

Checklist task 

Parameter task 

ALTITUDE 

3.2 

2.3 

CABIN ALTITUDE 

3.7 

5.4 

FUEL QTY 

4.8 

4.2 

ENG OIL PRESS 

5.4 

5.6 

FLAP/SLAT ASYMMETRY 

6.7 

8.3 

FUEL SYS PRESS 

6.9 

7.4 

HYD PUMP OVHT 

8.5 

11.0 

FLAP/SLAT DISAGREE 

9.1 

10.4 

HYD QTY 

9.4 

7.4 

HYD SYS PRESS 

9.4 

8.1 

GEN DRIVE 

9.4 

9.3 

FUEL PUMP PRESS 

9.6 

10.5 

FUEL CONFIG 

10.6 

8.5 

HYD PUMP PRESS 

11.8 

10.4 

MAIN BAT 

13.8 

13.7 

ENGBTL 

14.9 

15.5 

APU BAT 

16.6 

15.7 

APUBTL 

17.1 

17.3 


a The lower the number, the higher the ranking. 
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Table 8. Ratings of Predictive Information 



Predictive information 
(a) 

Question considerations 

Moving 

abnormally 

Rate of 
change 

Value at 
certain time 

Time to 
certain value 

General (question 3) 

86 

52 

61 

77 

Parameter 

System 

82 

60 

63 

74 

Navigation 

74 

60 

62 

75 

Flight control 

81 

75 

54 

63 

Fault development 

80 

70 

60 

74 

Phase of flight 

Taxi 

72 

57 

45 

47 

Take-off 

79 

66 

44 

50 

Climb 

84 

70 

57 

69 

Cruise 

83 

68 

65 

73 

Descent 

81 

61 

56 

68 

Approach 

78 

63 

47 

66 

Landing 

71 

54 

31 

42 

Time criticality 

<5 sec 

82 

57 

28 

44 

5-60 sec 

84 

67 

33 

57 

1-5 min 

82 

65 

50 

66 

5-15 min 

79 

66 

65 

73 

>15 min 

77 

63 

67 

77 

Aircraft 

Low speed 

77 

51 

52 

64 

Corporate jets 

84 

60 

56 

67 

Commercial jets 

83 

66 

56 

73 

>Mach 1 

88 

66 

57 

79 


*0 - low rating, 100 = high rating. 
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Table 9. Situation Awareness Ratings for Predictive Information 


Predictive information 

Situation awareness rating 
(a) 

Moving abnormally 

85 

Rate of change 

65 

Value at a certain time 

59 

Time to a certain value 

69 


a 0 = low rating, 100 = high rating. 
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Verbal briefing of research done 
at Langley Research Center 


Subject completes pilot 
background questionnaire 
(Appendix A) 


Subject signs informed consent form 
(Appendix B) 


Subject reads verbal briefing highlights and general pilot directions 
(Appendix C) 


Checklist paired-comparison task directions 
(no predictive information available) 


Parameter-Ordering Task 
(predictive capabilities) 


Survey 

(Various types of predictive 
information in certain conditions; 
Appendix G) 


Subject sees definition of checklists 
(Table 2) 

Subject sees checklist paired-comparison directions 
(Appendix D) 

Subject practices 
(Figure 2) 

Subject given Boeing checklists 
(Appendix E) 


Subject sees parameter-ordering task directions 
(Appendix F) 

Subject given cards 
(Table 3) 


Figure 1 . Schematic of procedure for experiment. 
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Pilot rating 
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